Training image enhancement model and enhancing image

ABSTRACT

Methods, devices, systems and apparatus for training image enhancement models and enhancing images are provided. In one aspect, a method of training an image enhancement model includes: for each of one or more constraint features, processing a ground truth image with the constraint feature to obtain a feature image corresponding to the constraint feature, for each of the one or more feature images, using the ground truth image and the feature image to train a convolutional neural network (CNN) structure model corresponding to the feature image, determining a loss function of the image enhancement model based on the one or more CNN structure models corresponding to the one or more feature images, and establishing the image enhancement model based on the loss function. A to-be-enhanced image can be input into the established image enhancement model to obtain an enhanced image.

CROSS REFERENCE TO RELATED APPLICATIONS

The present disclosure claims priority to Chinese Patent Application No.201910395631.0 filed on May 13, 2019, the entire content of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of image processing, andmore particularly, to methods and apparatus for training imageenhancement models and enhancing images.

BACKGROUND

As a data expression method, a neural network can approximate anyfunction in theory. The essence of training is to find a weightcorresponding to each layer of the neural network through a learningalgorithm to ensure that an output of the neural network can fit acorresponding input. A loss function can be used to quantify thisobjective.

In some cases, image enhancement processing may be carried out bymanually designing certain features for a loss function to achieve imageenhancement effects such as image de-noising, artifact reduction,de-blurring, and image restoration.

NEUSOFT MEDICAL SYSTEMS CO., LTD. (NMS), founded in 1998 with its worldheadquarters in China, is a leading supplier of medical equipment,medical IT solutions, and healthcare services. NMS supplies medicalequipment with a wide portfolio, including CT, Magnetic ResonanceImaging (MRI), digital X-ray machine, ultrasound, Positron EmissionTomography (PET), Linear Accelerator (LINAC), and biochemistry analyser.Currently, NMS' products are exported to over 60 countries and regionsaround the globe, serving more than 5,000 renowned customers. NMS'slatest successful developments, such as 128 Multi-Slice CT ScannerSystem, Superconducting MRI, LINAC, and PET products, have led China tobecome a global high-end medical equipment producer. As an integratedsupplier with extensive experience in large medical equipment, NMS hasbeen committed to the study of avoiding secondary potential harm causedby excessive X-ray irradiation to the subject during the CT scanningprocess.

SUMMARY

The present disclosure provides methods, devices, systems and apparatusfor training image enhancement models and enhancing images using thetrained image enhancement models.

One aspect of the present disclosure features a computer-implementedmethod of training image enhancement models. The computer-implementedmethod includes: for each of one or more constraint features, processinga ground truth image with the constraint feature to obtain a featureimage corresponding to the constraint feature; for each of one or morefeature images corresponding to the one or more constraint features,training a convolutional neural network (CNN) structure modelcorresponding to the feature image using the ground truth image and thefeature image; determining a loss function of an image enhancement modelbased on one or more CNN structure models corresponding to the one ormore feature images; and establishing the image enhancement model basedon the loss function.

In some embodiments, the one or more constraint features include atleast one of: a Sobel feature, a Prewitt feature, a contourlet transformfeature, a gradient feature, or a feature of a target layer in acorresponding CNN structure model, the feature of the target layer beingsuperior to the feature of each of other layers in the corresponding CNNstructure model.

In some embodiments, training the CNN structure model corresponding tothe feature image using the ground truth image and the feature imageincludes: training, based on deep learning, the CNN structure modelcorresponding to the feature image using the ground truth image as aninput image and the feature image as a label image.

In some embodiments, determining the loss function of the imageenhancement model based on the one or more CNN structure modelscorresponding to the one or more feature images includes: determining arespective weight value for each of the one or more CNN structuremodels; multiplying each of the one or more CNN structure models withthe respective weight value to obtain a respective product; and taking asum of the one or more respective products as the loss function of theimage enhancement model.

In some embodiments, determining the respective weight value of each ofthe one or more CNN structure models can includes: for each of the oneor more CNN structure models, determining the respective weight value ofthe CNN structure model based on an order of magnitude and acontribution corresponding to the CNN structure model, the respectiveweight value of the CNN structure model being proportional to thecontribution corresponding to the CNN structure model. For each of theone or more CNN structure models, a product of multiplying the order ofmagnitude corresponding to the CNN structure model with the respectiveweight value can be an identical target order of magnitude. The order ofmagnitude corresponding to the CNN structure model can be an order ofmagnitude of numerical values of a predicted image from the CNNstructure model, and the numerical values of the predicted image canrefer to pixel values in the predicted image.

In some embodiments, establishing the image enhancement model based onthe loss function includes: back propagating an error value computed bythe loss function to adjust parameter values for each layer in the imageenhancement model; and establishing the image enhancement model with theadjusted parameter values.

Another aspect of the present disclosure features a method of enhancingimages, including: obtaining a pre-established image enhancement modeland enhancing an image by inputting the image into the pre-establishedimage enhancement model to obtain an enhanced image. The pre-establishedenhancement model can be established based on a loss function. The lossfunction can be determined based on one or more convolutional neuralnetwork (CNN) structure models respectively associated with one or morefeature images. Each of the one or more CNN structure models can betrained using an associated feature image and a ground truth image, theassociated feature image being obtained by processing the ground truthimage with a corresponding constrain feature.

In some embodiments, for each of the one or more CNN structure models,the corresponding constraint feature includes one of: a Sobel feature, aPrewitt feature, a contourlet transform feature, a gradient feature, anda feature of a target layer in the CNN structure model, the feature ofthe target layer in being superior to the feature of each of otherlayers in the CNN structure model.

Each of the one or more CNN structure models can be trained based ondeep learning using the ground truth image as an input image and theassociated feature image as a label image. The loss function of thepre-established image enhancement model can be a sum of products, eachof the products being obtained by multiplying a corresponding one of theone or more CNN structure models with a weight value associated with theCNN structure model. The weight value associated with the CNN structuremodel can be determined based on an order of magnitude and acontribution corresponding to the CNN structure model, the weight valuebeing proportional to the contribution, and for each of the one or moreCNN structure models, a product of multiplying the order of magnitudecorresponding to the CNN structure model with the weight valueassociated with the CNN structure model can be an identical target orderof magnitude. The order of magnitude corresponding to the CNN structuremodel can be an order of magnitude of numerical values of a predictedimage from the CNN structure model, and the numerical values of thepredicted image cab refer to pixel values in the predicted image.

In some embodiments, the pre-established image enhancement model isestablished based on the loss function by back propagating an errorvalue computed by the loss function to adjust parameter values for eachlayer in the pre-established image enhancement model and establishingthe pre-established image enhancement model with the adjusted parametervalues.

A further aspect of the present disclosure features a device including:at least one processor; and at least one non-transitory machine readablestorage medium coupled to the at least one processor havingmachine-executable instructions stored thereon that, when executed bythe at least one processor, cause the at least one processor to performoperations including: for each of one or more constraint features,processing a ground truth image with the constraint feature to obtain afeature image corresponding to the constraint feature; for each of oneor more feature images corresponding to the one or more constraintfeatures, training a convolutional neural network (CNN) structure modelcorresponding to the feature image using the ground truth image and thefeature image; determining a loss function of the image enhancementmodel based on one or more CNN structure models corresponding to the oneor more feature images; and establishing the image enhancement modelbased on the loss function.

In some embodiments, the one or more constraint features include atleast one of: a Sobel feature, a Prewitt feature, a contourlet transformfeature, a gradient feature, or a feature of a target layer in acorresponding CNN structure model, the feature of the target layer beingsuperior to the feature of each of other layers in the corresponding CNNstructure model.

Training the CNN structure model corresponding to the feature imageusing the ground truth image and the feature image can include:training, based on deep learning, the CNN structure model correspondingto the feature image using the ground truth image as an input image andthe feature image as a label image. Determining the loss function of theimage enhancement model based on the one or more CNN structure modelscorresponding to the one or more feature images can include: determininga respective weight value of each of the one or more CNN structuremodels; multiplying each of the one or more CNN structure models withthe respective weight value to obtain a respective product; and taking asum of the one or more respective products as the loss function of theimage enhancement model.

In some embodiments, determining the respective weight value of each ofthe one or more CNN structure models includes: for each of the one ormore CNN structure models, determining a weight value of the CNNstructure model based on an order of magnitude and a contributioncorresponding to the CNN structure model, the weight value of the CNNstructure model being proportional to the contribution of the CNNstructure model. For each of the one or more CNN structure models, aproduct of multiplying the order of magnitude corresponding to the CNNstructure model with the weight value associated with the CNN structuremodel can be an identical target order of magnitude.

The operations can further include: inputting a to-be-enhanced image tothe image enhancement model to obtain an enhanced image.

The details of one or more examples of the subject matter described inthe present disclosure are set forth in the accompanying drawings anddescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims. Features of the present disclosure are illustrated byway of example and not limited in the following figures, in which likenumerals indicate like elements.

BRIEF DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are schematic diagrams illustrating an effect of imageenhancement.

FIG. 2 is a schematic diagram illustrating a scene of training an imageenhancement model.

FIG. 3 is a flowchart illustrating a process of a method of training animage enhancement model according to one or more examples of the presentdisclosure.

FIGS. 4A and 4B are schematic diagrams illustrating feature images totrain the image enhancement model according to one or more examples ofthe present disclosure.

FIG. 5 is a schematic diagram illustrating a scenario of training a CNNstructure model according to one or more examples example of the presentdisclosure.

FIG. 6 is a flowchart illustrating another method of training an imageenhancement model according to one or more examples of the presentdisclosure.

FIG. 7 is a schematic diagram illustrating an image enhancement modelaccording to one or more examples of the present disclosure.

FIGS. 8A to 8C are schematic diagrams illustrating image enhancementeffects according to one or more examples of the present disclosure.

FIG. 9 is a schematic diagram illustrating a hardware structure of anelectronic device according to one or more examples of the presentdisclosure.

DETAILED DESCRIPTION

Image enhancement processing may be carried out by manually designingcertain features for a loss function. If the loss function is providedrelatively simply, the loss function can be expressed in an analyticalexpression form. However, most of such loss functions treat each pixelin the image equally, generally considering only grayscale informationof the pixels, with no consideration on relationship betweenneighbouring pixels and location information of the pixels, which maycause problems such as over-blurring and over-smoothing in the resultedimage.

If the loss function is provided relatively complicatedly, it isdifficult to express such loss function in the analytical expressionform. Even if an analytical expression is obtained, it is difficult tosolve a derivative of the analytical expression.

In addition, considering that most of the loss functions include onlyone or two parts of the features, and cannot fully reflect multiplefeatures to be constrained. If the number of components is increased forthe loss function, on the one hand, the running time of the algorithmwill be prolonged, and on the other hand, the above problems caused bythe complicated loss function may occur.

Before describing the method of training an image enhancement modelprovided by examples of the present disclosure, an image enhancementprocessing process in the related art will be explained first.

Image enhancement can include a series of inverse derivation problems.Image enhancement can include image de-noising, artifact reduction,de-blurring, image recovery and other categories, which belongs to aprocess of seeking cause from an effect. The purpose is to find a set ofoptimal parameters to characterize a directly observed system, which isinvolved in many scientific fields and engineering fields and shown inthe following formula:

g=T(f _(true)+δ_(g))  (1)

Formula (1) indicates reconstructing signals f_(true)∈X from data g∈Y,X, Y respectively denote a vector space, δ_(g) denotes noise. A mappingrelationship Y→X can model a given signal with noise, to realize thespace conversion.

A priori knowledge can be used to constrain such inverse problems. Basedon the priori knowledge, it is possible to narrow the scope of thesolution space by constraining the unique features of the image from alarge number of solution spaces, which is conducive to finding a globaloptimal solution. Therefore, in this process, designing a strongregularization of the priori knowledge is a crucial step.

The priori knowledge of image enhancement can be manually designed basedon characteristics of the image and experience. For example, most ofnatural images show sparseness and piecewise smoothing after wavelettransform. In this case, the priori knowledge includes a constraint of awavelet coefficient or that the gradient space has sparseness. Althoughthese priori knowledge can be used in the field of image enhancement, ifthe loss function is constructed based on the priori knowledge and theconstructed loss function is then used to enhance the image, it is easyto cause over-smoothing in the enhanced image because the optimalsolution of the loss function cannot be constrained. For example, whenthe image shown in FIG. 1A is subjected to image enhancement processingto obtain the enhanced image shown in FIG. 1B, the enhanced image shownin FIG. 1B has a problem of over-smoothing. In some cases, to solve theproblem of over-smoothing caused by the loss function constructed withthe priori knowledge, some feature parameters of the loss function canbe highlighted to achieve different image enhancement effects.

For example, to achieve the image enhancement effect that highlights theimage edge, the norm L₁ can be weighted when designing the lossfunction, as shown in the following formula:

argmin_(x) E(x)=α∥Ax−y∥ ₁  (2),

where α denotes a weight coefficient, and for an edge area with arelatively larger gradient, the weight coefficient can be set to arelatively larger value. In contrast, for a relatively flat area, theweight coefficient can be set to a relatively small value. x denotes aninput image, y denotes a ground truth image, and A denotes a matrixcorresponding to the current algorithm. For example, when the currentalgorithm is an image enhancement algorithm, A denotes an imageenhancement matrix; when the current algorithm is the image de-noisingalgorithm, A denotes an image de-noising matrix. 1 denotes the norm L₁,and arg min denotes a variable value that makes the objective functiontake the minimum value.

The configuration of the above algorithm is shown in FIG. 2. Ato-be-enhanced image is input into a network structure Network_(inverse)to obtain a predicted image, and then the predicted image together withthe ground truth image corresponding to the to-be-enhanced image isinput into a loss function to calculate an error. Inverse derivation isperformed from this error, to adjust respective parameter(s) in eachconvolution layer of Network_(inverse), so as to reduce subsequentlycalculated errors. In examples of the present disclosure, the term“convolution layer” and the term “layer” can be used interchangeably.

When the subsequently calculated errors are small enough, it can beconsidered that the training of the network structure Network_(inverse)is completed. Then, a to-be-enhanced image may be input the trainednetwork structure Network_(inverse), through forward propagation in thetrained network structure Network_(inverse), a predicted image can beobtained. At this time, the predicted image is an enhanced image.

However, the loss function involved in the above method is either toosimple, which may cause problems such as over-blurring andover-smoothing in the image, or too complicated and cannot be expressedin an analytical expression form, which may make derivation difficult.Moreover, the loss function cannot fully reflect multiple constraintfeatures.

Based on this, embodiments of the present disclosure provide a method oftraining an image enhancement model. FIG. 3 is a flowchart illustratinga process of a method of training an image enhancement model accordingto one or more examples of the present disclosure. As shown in FIG. 3,the process includes the following steps 101-104.

At step 101, for each of one or more constraint features, a ground truthimage is processed by the constraint feature to obtain a feature imagecorresponding to the constraint feature.

At step 102, for each of the one or more feature images corresponding tothe one or more constrain features, the ground truth image and thefeature image are used to train a convolutional neural network (CNN)structure model corresponding to the feature image.

At step 103, a loss function of the image enhancement model isdetermined based on the CNN structure models corresponding to the one ormore feature images.

At step 104, the image enhancement model is established based on theloss function, e.g., by training the image enhancement model based onthe loss function as described below.

Based on steps 101-104, for each constraint feature, a corresponding CNNstructure model can be trained separately; based on all of the CNNstructure models, the loss function of the image enhancement model canbe obtained; and finally the image enhancement model can be establishedbased on the loss function. Through the above process, the loss functionis no longer expressed in an analytical expression manner, but ismodelled, which can solve the problems of over-smoothing andover-blurring in the image enhancement process, and speed up the imageenhancement processing.

For the above step 101, relevant techniques can be used to extract atleast one constraint feature (or one or more constraint features) fromthe ground truth image. In some examples, assume that there are twoconstraint features: a first constraint feature and a second constraintfeature. The first constraint feature is extracted from the ground truthimage to obtain a feature image corresponding to the first constraintfeature. The second constraint feature is extracted from the groundtruth image to obtain a feature image corresponding to the secondconstraint feature. The ground truth image in examples of the presentdisclosure can refer to a high-quality image corresponding to an inputimage. The ground truth image is a desired or expected image which canbe obtained when the image enhancement model works perfectly. Forexample, in image de-noising, an input for a deep learning network is animage with noise, and the ground truth image thereof refers to ahigh-quality image corresponding to the image with noise, that is, animage without noise. The ground truth image has the same shape as thatof the input image. The ground truth image involved in examples of thepresent disclosure include but not limited to a two-dimensional image.

The at least one constraint feature can include any image feature, suchas a Sobel feature and a Pruitt feature. Sobel operator is a discretefirst-order differentiation operator and used to compute anapproximation of the gradient of the image intensity function. ThePrewitt operator is a discrete differentiation operator and used tocompute an approximation of the gradient of the image intensityfunction. At each point in the image, the result of the Prewitt operatoris either the corresponding gradient vector or the norm of this vector.The Prewitt operator is based on convolving the image with a small,separable, and integer valued filter in horizontal and verticaldirections and is therefore relatively inexpensive in terms ofcomputations like Sobel and Kayyali operators. On the other hand, thegradient approximation which it produces is relatively crude, inparticular for high frequency variations in the image. The Sobel featureand the Prewitt operator are used with edge detection algorithms.

The at least one constraint feature can also include a contourlettransform feature. The contourlet transform is a multiresolution, local,and directional two-dimensional image representation method. Thecontourlet transform performs multi-scale analysis and directionanalysis separately. That is, first, LP (LaplacianPyramid) transform isperformed to decompose the image in multiple dimensions, to obtainsingular points; and then, singular points distributed in the samedirection are combined into a coefficient through a DFB (DirectionalFilter Bank) to approximate the original image with a basis functionsimilar to the contour segment. The length of a support interval of thebasis function changes with the scale, and can describe the edge of theimage in a near optimal way. The contourlet transform is defineddirectly in a discrete domain, and then the discrete domain and thecontinuous domain are connected, and its approximation is discussed inthe continuous domain.

The at least one constraint feature can also include a gradient feature.An image may be interpreted as a two-dimensional discrete number set. Bygeneralizing a two-dimensional continuous method to find the partialderivative of the function, the partial derivative of the image isobtained, that is, the maximum rate of change at that point, which isthe gradient feature herein.

In some examples, the at least one constraint feature also includes afeature of a target layer in a CNN structure model, and the feature ofthe target layer is superior to the feature of each of other layers inthe CNN structure model. Neural networks can include dozens or evenhundreds of layers, and the output of each layer of the CNN structuremodel can be regarded as a feature. For example, if an edge displayfeature of the target layer in the CNN structure model trained in othersimilar tasks is superior to the edge display features of the otherlayers, the edge display feature of the target layer can be used as theconstraint feature. Edge feature include edge display feature. Edgefeature can highlight the edges of the image. In other words, the edgeof the image can be obtained by using edge feature. During training ofthe image enhancement model, at least one of the above-mentionedconstrain features can be extracted from the ground truth imagerespectively, so as to obtain a feature image corresponding to each ofthe constrain features. For example, different constrain features can berespectively extracted from the ground truth image corresponding to theimage shown in FIG. 1A, to obtain feature images shown in FIGS. 4A and4B. The feature image shown in FIG. 4A is a feature image obtained byprocessing the ground truth image with a filter feature, and the featureimage shown in FIG. 4B is a feature image obtained by processing theground truth image with an edge feature. In some examples, Gaussianfiltering, median filtering or other operations can be used to processthe ground truth image to obtain the feature image corresponding to thefilter feature shown in FIG. 4A.

For the step 102, when training the CNN structure models correspondingto the feature images, as shown in FIG. 5, the ground truth image isused as an input image, and each of the feature images can be used as alabel image. Each CNN structure model can include a respective Networkstructure Network_(fi), where i is an integer among a range from 1 to nand Networkf_(i) denotes the i-th CNN structure model. Each networkstructure Network_(fi) can include a series of convolution layers andactivation functions between adjacent convolution layers. In FIG. 5,taking the network structure Network_(fi) as an example, the networkstructure Network_(fi) including a series of convolution layers andactivation functions between adjacent convolution layers is shown.Further, the respective CNN structure model Network_(f1), Network_(f2),. . . , Network_(fn), corresponding to each of the feature images istrained based on depth learning. The label image has the same size asthat of the input image.

For the step 103, as shown in FIG. 6, FIG. 6 is a flowchart illustratinga process of a method of training an image enhancement model shown onthe basis of the example shown in FIG. 1. The process includes the abovesteps 101-104 and related details are omitted for brevity. Step 103includes the following steps.

At step 103-1, a respective weight value of each of the CNN structuremodels is determined.

At this step 103-1, for each CNN structure model, a weight value of theCNN structure model can be determined based on an order of magnitude anda contribution corresponding to the CNN structure model. The order ofmagnitude corresponding to the CNN structure model is an order ofmagnitude of numerical values of a predicted image from the CNNstructure model, and the numerical values of the predicted image referto pixel values in the predicted image, such as grey values. For each ofthe CNN structure models, an identical target order of magnitude isobtained by multiplying the order of magnitude corresponding to the CNNstructure model with the weight value corresponding to the CNN structuremodel. That is, for each of the CNN structure models, the product ofmultiplying the corresponding order of magnitude with the correspondingweight value is identical to each other. The weight value for the CNNstructure model is proportional to the contribution for the CNNstructure model. The contribution for the CNN structure model depends ona desired image enhancement effect. For example, if the desired imageenhancement effect is to increase clarity of the entire image, theweight value of the CNN structure model associated with the clarity ofthe entire image is relatively larger. For another example, if thedesired image enhancement effect is to increase clarity of the outlineof the image, the weight value of the CNN structure model associatedwith the clarity of the outline of the image is relatively larger. Foranother example, when the desired image enhancement effect is to retainsmall structures in the image as much as possible, the weight value ofthe CNN structure model associated with the retention degree of smallstructures is relative larger.

For example, the order of magnitude of the numerical values in thepredicted image from the first CNN structure model is 10², such as from100 to 500, and the order of magnitude of the numerical values in thepredicted image from the second CNN structure model is 10⁻¹, such asfrom 0.1 to 0.5. When determining the loss function subsequently, toavoid that since the order of magnitude of the numerical values in thepredicted image from the first CNN structure model significantly differsfrom the order of magnitude of the numerical values in the predictedimage from the second CNN structure model, the second CNN structuremodel is weaker than the first CNN structure model in the loss function,the order of magnitude of the weight value of the first CNN structuremodel may be set to 10⁻², and the order of magnitude of the weight valueof the second CNN structure model may be set to 10¹. In this way, anidentical target order of magnitude can be obtained by multiplying therespective order of magnitude corresponding to each CNN structure modelwith a respective weight value.

In addition, the weight value of the CNN structure model may bedetermined based on the contribution of the CNN structure model, thatis, the weight value of the CNN structure model may be determined basedon the contribution of the constraint feature corresponding to the CNNstructure model. For example, if the desired image enhancement effect isto obtain an enhanced image with more clarified edges, the weight valueof the CNN structure model corresponding to the edge feature can be setlarger than the weight values of the CNN structure models correspondingto other constraint features. In this way, an enhanced image with moreclarified edges can be obtained based on the loss function.

At step 103-2, a sum of products each of which is obtained bymultiplying the respective CNN structure model with the respectiveweight value is used as the loss function of the image enhancementmodel.

In an example of the present disclosure, the loss function Loss of theimage enhancement model can be expressed by the following formula:

$\begin{matrix}{{{Loss} = {\sum\limits_{i = 1}^{n}{a_{i} \times {Networkf}_{i}}}},} & (3)\end{matrix}$

where Networkf_(i) denotes the i-th CNN structure model, and a_(i)denotes the weight value corresponding to the i-th CNN structure model.In some examples, after determining the loss function, an optimizationalgorithm can be used to determine a solution to reach a minimum valueof the loss function.

For the step 104, the process of establishing an image enhancement modelbased on the loss function can be as shown in FIG. 7.

As illustrated in FIG. 7, the image enhancement model can include anetwork structure Network_(inverse). The network structureNetwork_(inverse) can include a series of convolution layers withactivation functions between adjacent convolution layers. Ato-be-enhanced image can be input into the network structureNetwork_(inverse) to obtain a predicted image, and then the predictedimage together with the ground truth image corresponding to theto-be-enhanced image can be input into a self-defined loss function tocalculate an error value. The self-defined loss function can bedetermined at step 103 in FIG. 3 or step 103-2 in FIG. 6. Theself-defined loss function Loss can be obtained based on loss functions(Loss1, Loss 2, . . . , Loss n) of the multiple CNN structure model(Network_(f1), Network_(f2), . . . , Network_(fn)), for example,according to the formula (3). For optimization, inverse derivation canbe performed from the error value to adjust respective parameter valuesin each convolution layer of Network_(inverse), so as to reducesubsequently calculated error values. When the subsequently calculatederror values are small enough, it can be considered that the training ofthe network structure Network_(inverse) (or the training of the imageenhancement model) is completed, or the image enhancement model isestablished. That is, after the loss function is determined, the errorvalue computed by the loss function can be back propagated to adjust theparameter values corresponding to each layer in the image enhancementmodel (or the network structure Network_(inverse)), thereby establishingthe image enhancement model with the adjusted parameter values.

For a complicated loss function, the training method provided by theexamples of the present disclosure does not need to determine ananalytical expression of the loss function, nor does it require aself-defined gradient calculation formula, but utilizes the automaticderivation function of the deep learning framework to implement aminimizing process of the loss function. In addition, the trainingmethod provided by the examples of the present disclosure canself-define a respective CNN structure model corresponding to multipleconstraint features, and take all of the multiple CNN structure modelsas components of the loss function of the image enhancement modelthrough weighted summation. In addition, the training method provided bythe examples of the present disclosure applies multiple constrains tomultiple solution spaces such that the optimal solution that meetsspecific tasks and specific requirements can be found.

In some examples of the present disclosure, based on the trained imageenhancement model, a process of a method of enhancing an image is alsoprovided. The process includes: inputting a to-be-enhanced image into apre-established image enhancement model to obtain an enhanced image.

The image enhancement model is a model obtained by using the abovemethod of training an image enhancement model.

As shown in FIG. 7, the to-be-enhanced image can be input into the imageenhancement model, and the enhanced image can be obtained throughforward propagation.

In the process of obtaining the loss function of the image enhancementmodel, the solution spaces have been constrained by different constraintfeatures, such as edge enhancement, de-blurring, etc. Therefore, theenhanced image obtained by the above method also has the constraintfeatures, so the edges of the enhanced image can be more clarified andhave higher resolution, and have no over-smoothing.

The method of enhancing an image provided by examples of the presentdisclosure includes two stages: an image enhancement model trainingstage and an image enhancement model application stage. The lossfunction is determined in the image enhancement model training stage.The determined loss function may be directly used in the imageenhancement model application stage, that is, there is no need tore-calculate the loss function in the image enhancement modelapplication stage, and thus the image enhancement model applicationstage runs faster. In other words, in the method of enhancing an imageprovided by examples of the present disclosure, a to-be-enhanced imageis input into the image enhancement model, and an enhanced image can bedirectly output, and thus the image enhancement model application runsfaster. The method of enhancing an image provided by examples of thepresent disclosure can be applied to any image enhancement processingsuch as image de-noising, image de-blurring, image restoration, andsuper-resolution reconstruction. When determining the loss function,different positions can be given different weight values to achieve amore comprehensive retention of high-frequency information in theresulted image and more clarified edges.

The examples of the present disclosure have been verified throughexperiments, taking low-dose CT image de-noising as an example. Ofcourse, it is noted that the application field of the disclosed methodis not limited to low-dose CT image de-noising, but is applicable to anyimage enhancement processing.

FIG. 8A shows an input low-dose CT image to be de-noised, FIG. 8B showsa corresponding ground truth image, and FIG. 8C shows an enhanced imagefor the low-dose CT image to be de-noised shown in FIG. 8A. The enhancedimage is obtained with the method of enhancing an image provided byexamples of the present disclosure. Compared with FIG. 1A and FIG. 1B,the experimental result obtained with the method of enhancing an imageprovided by examples of the present disclosure shows that the enhancedimage shown in FIG. 8C has better uniformity and lower noise, theclarity of the edges is not degraded, that is, the enhanced has noover-smoothing, and the details of the anatomical structure of theenhanced image are better preserved, which can meet the clinicalapplication needs and has good practicality.

The method of training an image enhancement model, especially the methodof training a loss function and the method of enhancing an image, can beuniformly executed by an electronic device, and the structure of theelectronic device can refer to the schematic diagram shown in FIG. 9. Asshown in FIG. 9, the electronic device includes a processor 910, acommunication interface 920, a memory 930, and a bus 940. The processor910, the communication interface 920, and the memory 930 implementcommunication with each other via the bus 940.

The memory 930 can store logic instructions for training an imageenhancement model and logic instructions for enhancing an image, and thememory can be, for example, a non-transitory memory. The processor 910can invoke and execute the logic instructions for training an imageenhancement model in the memory 930, to perform training first, obtainan image enhancement model, and then execute the logic instructions forenhancing an image. For example, the logic instructions for training animage enhancement model and the logic instructions for enhancing animage can be programs corresponding to some functions of the controlsoftware of a medical image acquisition system. When the processorexecutes the instructions, the electronic device can correspondinglydisplay a function interface corresponding to the instructions on thedisplay interface.

The functions of the logic instructions for training an imageenhancement model and the logic instructions for enhancing an image canbe stored in a non-transitory computer-readable storage medium if theyare implemented in the form of software functional units and sold orused as independent products. Based on such an understanding, thetechnical solution of the present disclosure essentially or with thepart contributing to the existing technology or part of the technicalsolution can be embodied in the form of a software product, the computersoftware product is stored in a storage medium, including someinstructions are used to cause a computer device (which can be apersonal computer, a server, or a network device, etc.) to perform allor part of the steps of the methods described in the embodiments of thepresent disclosure. The storage medium includes a U disk, a mobile harddisk, a read-only memory (ROM), a random access memory (RAM), a magneticdisk or an optical disk and other medium that can store program codes.

For simplicity and illustrative purposes, the present disclosure isdescribed by referring mainly to examples thereof. In the abovedescriptions, numerous specific details are set forth in order toprovide a thorough understanding of the present disclosure. It will bereadily apparent however, that the present disclosure may be practicedwithout limitation to these specific details. In other instances, somemethods and structures have not been described in detail so as not tounnecessarily obscure the present disclosure. As used herein, the terms“a” and “an” are intended to denote at least one of a particularelement, the term “includes” means includes but not limited to, the term“including” means including but not limited to, and the term “based on”means based at least in part on.

The above description is merely preferred examples of the presentdisclosure and is not intended to limit the present disclosure in anyform. Although the present disclosure is disclosed by the aboveexamples, the examples are not intended to limit the present disclosure.Those skilled in the art, without departing from the scope of thetechnical scheme of the present disclosure, may make a plurality ofchanges and modifications of the technical scheme of the presentdisclosure by the method and technical content disclosed above.

Therefore, without departing from the scope of the technical scheme ofthe present disclosure, based on technical essences of the presentdisclosure, any simple alterations, equal changes and modificationsshould fall within the protection scope of the technical scheme of thepresent disclosure. Accordingly, other embodiments are within the scopeof the following claims.

What is claimed is:
 1. A computer-implemented method of training imageenhancement models, the computer-implemented method comprising: for eachof one or more constraint features, processing a ground truth image withthe constraint feature to obtain a feature image corresponding to theconstraint feature; for each of one or more feature images correspondingto the one or more constraint features, training a convolutional neuralnetwork (CNN) structure model corresponding to the feature image usingthe ground truth image and the feature image; determining a lossfunction of an image enhancement model based on one or more CNNstructure models corresponding to the one or more feature images; andestablishing the image enhancement model based on the loss function. 2.The computer-implemented method of claim 1, wherein the one or moreconstraint features comprise at least one of: a Sobel feature, a Prewittfeature, a contourlet transform feature, a gradient feature, or afeature of a target layer in a corresponding CNN structure model, thefeature of the target layer being superior to the feature of each ofother layers in the corresponding CNN structure model.
 3. Thecomputer-implemented method of claim 1, wherein training the CNNstructure model corresponding to the feature image using the groundtruth image and the feature image comprises: training, based on deeplearning, the CNN structure model corresponding to the feature imageusing the ground truth image as an input image and the feature image asa label image.
 4. The computer-implemented method of claim 1, whereindetermining the loss function of the image enhancement model based onthe one or more CNN structure models corresponding to the one or morefeature images comprises: determining a respective weight value for eachof the one or more CNN structure models; multiplying each of the one ormore CNN structure models with the respective weight value to obtain arespective product; and taking a sum of the one or more respectiveproducts as the loss function of the image enhancement model.
 5. Thecomputer-implemented method of claim 4, wherein determining therespective weight value of each of the one or more CNN structure modelscomprises: for each of the one or more CNN structure models, determiningthe respective weight value of the CNN structure model based on an orderof magnitude and a contribution corresponding to the CNN structuremodel, the respective weight value of the CNN structure model beingproportional to the contribution corresponding to the CNN structuremodel, wherein, for each of the one or more CNN structure models, aproduct of multiplying the order of magnitude corresponding to the CNNstructure model with the respective weight value is an identical targetorder of magnitude.
 6. The computer-implemented method of claim 5,wherein the order of magnitude corresponding to the CNN structure modelis an order of magnitude of numerical values of a predicted image fromthe CNN structure model, and wherein the numerical values of thepredicted image refer to pixel values in the predicted image.
 7. Thecomputer-implemented method of claim 1, wherein establishing the imageenhancement model based on the loss function comprises: back propagatingan error value computed by the loss function to adjust parameter valuesfor each layer in the image enhancement model; and establishing theimage enhancement model with the adjusted parameter values.
 8. A methodof enhancing images, comprising: obtaining a pre-established imageenhancement model; and enhancing an image by inputting the image intothe pre-established image enhancement model to obtain an enhanced image,wherein the pre-established enhancement model is established based on aloss function, wherein the loss function is determined based on one ormore convolutional neural network (CNN) structure models respectivelyassociated with one or more feature images, and wherein each of the oneor more CNN structure models is trained using an associated featureimage and a ground truth image, the associated feature image beingobtained by processing the ground truth image with a correspondingconstrain feature.
 9. The method of claim 8, wherein, for each of theone or more CNN structure models, the corresponding constraint featurecomprises one of: a Sobel feature, a Prewitt feature, a contourlettransform feature, a gradient feature, and a feature of a target layerin the CNN structure model, the feature of the target layer in beingsuperior to the feature of each of other layers in the CNN structuremodel.
 10. The method of claim 8, wherein each of the one or more CNNstructure models is trained based on deep learning using the groundtruth image as an input image and the associated feature image as alabel image.
 11. The method of claim 8, wherein the loss function of thepre-established image enhancement model is a sum of products, each ofthe products being obtained by multiplying a corresponding one of theone or more CNN structure models with a weight value associated with theCNN structure model.
 12. The method of claim 11, wherein the weightvalue associated with the CNN structure model is determined based on anorder of magnitude and a contribution corresponding to the CNN structuremodel, the weight value being proportional to the contribution, andwherein, for each of the one or more CNN structure models, a product ofmultiplying the order of magnitude corresponding to the CNN structuremodel with the weight value associated with the CNN structure model isan identical target order of magnitude.
 13. The method of claim 12,wherein the order of magnitude corresponding to the CNN structure modelis an order of magnitude of numerical values of a predicted image fromthe CNN structure model, and the numerical values of the predicted imagerefer to pixel values in the predicted image.
 14. The method of claim 8,wherein the pre-established image enhancement model is established basedon the loss function by back propagating an error value computed by theloss function to adjust parameter values for each layer in thepre-established image enhancement model, and establishing thepre-established image enhancement model with the adjusted parametervalues.
 15. A device comprising: at least one processor; and at leastone non-transitory machine readable storage medium coupled to the atleast one processor having machine-executable instructions storedthereon that, when executed by the at least one processor, cause the atleast one processor to perform operations comprising: for each of one ormore constraint features, processing a ground truth image with theconstraint feature to obtain a feature image corresponding to theconstraint feature; for each of one or more feature images correspondingto the one or more constraint features, training a convolutional neuralnetwork (CNN) structure model corresponding to the feature image usingthe ground truth image and the feature image; determining a lossfunction of the image enhancement model based on one or more CNNstructure models corresponding to the one or more feature images; andestablishing the image enhancement model based on the loss function. 16.The device of claim 15, wherein the one or more constraint featurescomprise at least one of: a Sobel feature, a Prewitt feature, acontourlet transform feature, a gradient feature, or a feature of atarget layer in a corresponding CNN structure model, the feature of thetarget layer being superior to the feature of each of other layers inthe corresponding CNN structure model.
 17. The device of claim 15,wherein training the CNN structure model corresponding to the featureimage using the ground truth image and the feature image comprises:training, based on deep learning, the CNN structure model correspondingto the feature image using the ground truth image as an input image andthe feature image as a label image.
 18. The device of claim 15, whereindetermining the loss function of the image enhancement model based onthe one or more CNN structure models corresponding to the one or morefeature images comprises: determining a respective weight value of eachof the one or more CNN structure models; multiplying each of the one ormore CNN structure models with the respective weight value to obtain arespective product; and taking a sum of the one or more respectiveproducts as the loss function of the image enhancement model.
 19. Thedevice of claim 18, wherein determining the respective weight value ofeach of the one or more CNN structure models comprises: for each of theone or more CNN structure models, determining a weight value of the CNNstructure model based on an order of magnitude and a contributioncorresponding to the CNN structure model, the weight value of the CNNstructure model being proportional to the contribution of the CNNstructure model, wherein, for each of the one or more CNN structuremodels, a product of multiplying the order of magnitude corresponding tothe CNN structure model with the weight value associated with the CNNstructure model is an identical target order of magnitude.
 20. Thedevice of claim 15, wherein the operations further comprise: inputting ato-be-enhanced image to the image enhancement model to obtain anenhanced image.